Hi! My name is Chuan Yan and
I’ll introduce our work FlatMagic: Improving Flat Colorization
through AI-driven Design for Digital Comic Professionals.
This work is a collaboration with Yotam
Gingold, Ray Hong from George Mason University, John Chung, Eytan Adar
from University of Michigan and Kiheon
Yoon from Pusan National University.
The digitized drawing technology has greatly improved
the artist’s work efficiency.
However, their workload may still not easy as industry standards improved.
For example, the rapidly growing web comic has a much higher degree
of colorization, which even bring greater workload than before.
On the other hand, there are also many research works that
try to automate the colorization stage and many of them show impressive results.
However, few of those technologies have been widely used by professional artists.
Therefore, to figure out the reason behind, we interviewed five professionals
who have published their work in commercial comic distribution
platforms for more than three years And we aiming to understand:
their workflow, challenges in their work, general
thoughts and expecting features on AI tools,
and the key factors that affect them adopting AI automation.
The interview result shows there exists a common workflow, which include:
Line drawing, Flat colorization, Shading, Lighting, and Special Effects.
Additional analysis identified one special stage in this workflow
- Flatting, which shows several characteristics that particularly fit for AI-driven automation.
We will discuss the reasoning behind in a few next slides.
First of all, what is flatting? Flatting is a colorization stage
where the artists create a set of mutually exclusive layers.
And there exists
two industry requirements: consistency and completeness.
Flat consistency means the desired color regions are consistently correct.
they should always neither contain mergeable regions
nor spill out to their neighbor regions.
Therefore, professionals need to find and fix the falsely closed and falsely
opened lines before they bucket filling, which need them
to pay full attention.
The flat completeness requires the flat results only contain
mutual exclusive,
or no overlapping regions that cover the whole drawing space. Common failure
cases in the requirement are the “dirty bits” caused by anti-aliasing
pixels and gaps under line drawings when use the naive bucketing tools.
And fixing those problems is usually tedious and time consuming.
On the other side, participants also mentioned “lack of control” is
one of the issues that keep them from adopting the AI tools.
Because the output of AI usually can’t exactly match professional’s target.
However, it is difficult or even impossible to adjust.
Trying to fix those outputs will even need more effort than they
colorize from scratch. It is also difficult for AI tools
to find out the professional’s idea work in their mind.
The result will be highly personalized and there always exists
multi targets for each single input.
To sum up, we found that, different from other colorization
stages, flatting has only one “ground truth” for each line
drawing. Meanwhile, it is also a bottleneck in the workflow
since its costly workload when using the general drawing tools.
So any speed up on this stage can help professionals invest
more resources into later stages that matter more to the overall product
quality. Therefore, they are more open to let automate tools
deal with this less creative stage
So, we designed the FlatMagic. Given
an input line drawing, how to design an efficient flatting method?
We have discussed that professionals usually face artifacts
such as wrong flat region and “dirty bits” by naive bucket tools.
Therefore, it
is natural to build the interface of FlatMagic
around this bucketing interaction to let user bucket freely
and get a most correct result. Because flatting is color agnostic.
We built a neural network to simplify the input lines.
The target
of the simplification is created by extracting 1-pixel width boundary
from the idea flat result, We called this procedure
as neural re-line. Then we trained the neural re-line model based on a ground
truth dataset shared with us by a digital comic company.
Then fill the simplified lines
with different labels, and post process it to initial flatting result.
The user interface is implemented as an Adobe Photoshop plugin.
It receives the result from the neural reline
model, and allow users to manually colorize it
to the desired professional level result by our improved bucketing tool.
To verify the efficiency of our FlatMagic.
We recruited 16 students who are majoring digital
comic and animation, then asked them to flat 12 line
drawings by their best practice and by FlatMagic, respectively.
The flat time distribution result shows using FlatMagic helped participants
to significantly reduce their labor. And the outcomes
quality measurement shows participants could achieving similar or even slight
better outcomes to their best practice by using our FlatMagic.
So, here are some interesting high-level insights on our finding
which we called it intermediate representations.
Most of the AI’s automation
methods provide a full automation with no revisable check points, while
most professionals did not like this, which we call it AI over-scoping.
In contrast, intermediate representations offer a modularized architecture
and representation so that makes it possible for users
to know where and how to make necessary corrections.
So, we reflect on some possible decision points
when considering the design of AI systems.
First, we need to understand the detailed, stage by stage
workflow of professional’s task. Then carefully evaluate the degree
of AI usefulness and User intention at each stage in the workflow.
AI usefulness reflects how much work labor AI could cut down for the user.
User intention means how much a user desires
to let a particular step automated When consider
AI-usefulness and user intention as two different dimensions.
We can project all the AI automation methods on to the strategic space
for considering how and if this tool may be adopted.
The most promising case would be S1, where the AI is useful
and its valuation is positive. In S2, when the AI is useful,
we may still have a situation where an individual either doesn’t want
or doesn’t think they want, the automation. This suggests us that
it is worth considering a better adoption strategies to convince a user to adopt.
In S3, When the end-user may want something automated,
but the AI is unable to meet the object. Introducing a problematic
AI will discourage adopting the current or even all other similar tools.
This suggest us maybe finding a smaller intermediate representation
that gives a more reasonable AI-automation scope is needed.
In S4, while this worst case may be a “back to the drawing board”
situation, However, it may also could be the result of didn't correctly model
the needs, didn't build the right UI or did
not implement the algorithm well.
After we decide the right AI-automation scoping, we need to be careful
of merging the AI driven automation scope when build up the tool.
Cause merging multiple steps could ‘break’ a user’s mental model of the process.
The user may not be able to identify how or why
errors are occurring or how to correct for them.
At last, I want to thank all my collaborators
and Thank you for listening. Feel free to reach me out if you have any questions.